Method for categorizing open-ended comments

ABSTRACT

Some embodiments include a novel method for categorizing open-ended comments. The method relies on a person who originally made a comment to put the comment into a category. In some embodiments, the method receives a comment in response to an open-ended question, displays a set of categories that have been evaluated and determined to be likely to match a category for the received comment, receives a respondent-determined category for the comment, and categorizes the comment based on the respondent-determined category. This captures the category immediately after the comment is made with a high degree of accuracy.

BACKGROUND

Embodiments of the invention described in this specification relategenerally to categorization, and more particularly, to categorization ofopen-ended comments.

When a survey sponsor provides a forum for commenting, typically thesurvey sponsor is interested in identifying the types of comments beingmade in the forum. Generally, survey sponsors will identify types ofcomments through one or more categorization methods. However,categorizing comments is a time-consuming and expensive process. Todate, the existing methods of categorizing comments have been errorprone. One method requires survey sponsors to read comments andcategorize them. This method is tedious, time-consuming and expensivebecause each comment must be individually read and categorized.Furthermore, because a person who reads a comment on behalf of thesurvey sponsor is typically not the person who conceived of or wrote thecomment, the person's categorization of the comment can be inaccurate.Other methods are driven by lower-cost but complex algorithms whichsometimes make errors.

What is needed is a way to accurately and immediately categorizecomments.

BRIEF DESCRIPTION

Some embodiments of the invention include a novel method forcategorizing open-ended comments. The method relies on a person whooriginally made a comment to put the comment into a category. In thisway, the method captures the category immediately after the comment ismade with a high degree of accuracy.

In some embodiments, the method receives a comment in response to anopen-ended question, displays a set of categories that have beenevaluated and determined to be likely to match a category for thereceived comment, receives a respondent-determined category for thecomment, and categorizes the comment based on the respondent-determinedcategory.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in thisspecification. The Detailed Description that follows and the Drawingsthat are referred to in the Detailed Description will further describethe embodiments described in the Summary as well as other embodiments.Accordingly, to understand all the embodiments described by thisdocument, a full review of the Summary, Detailed Description, andDrawings is needed. Moreover, the claimed subject matters are not to belimited by the illustrative details in the Summary, DetailedDescription, and Drawings, but rather are to be defined by the appendedclaims, because the claimed subject matter can be embodied in otherspecific forms without departing from the spirit of the subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Having described the invention in general terms, reference is now madeto the accompanying drawings, which are not necessarily drawn to scale,and wherein:

FIG. 1 conceptually illustrates a process in some embodiments forcategorizing open-ended comments.

FIG. 2 conceptually illustrates a high level schematic view of a systemarchitecture that facilitates categorizing comments in some embodiments.

FIG. 3 conceptually illustrates a detailed schematic view of a systemarchitecture that facilitates categorizing comments in some embodiments.

FIG. 4 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are described.However, it will be clear and apparent to one skilled in the art thatthe invention is not limited to the embodiments set forth and that theinvention can be adapted for any of several applications.

Some embodiments of the invention include a novel method forcategorizing open-ended comments. The method relies on a person whooriginally made a comment to put the comment into a category. In thisway, the method captures the category immediately after the comment ismade with a high degree of accuracy.

In some embodiments, the method receives a comment in response to anopen-ended question, displays a set of categories that have beenevaluated and determined to be likely to match a category for thereceived comment, receives a respondent-determined category for thecomment, and categorizes the comment based on the respondent-determinedcategory.

In some embodiments, the method is implemented as a software applicationthat includes sets of instructions for performing the steps of themethod when the software application is running on a processor of acomputing device. In some embodiments, the software application accessesone or more computing devices over the Internet. In some embodiments,the software application connects to a web application server thataccesses one or more social media websites and a persistent datastorage. In some embodiments, the persistent data storage is a databaseaccessible through a database management system running on the processorof the computing device on which the software application is running. Insome embodiments, the software application is a client-side applicationthat accesses a cloud database management system.

As stated above, categorizing comments is a time-consuming and expensiveprocess which is prone to errors. Embodiments of the invention describedin this specification solve such problems by allowing survey respondentsto classify their own comments into categories.

The embodiments described in this specification differ from and improveupon currently existing options. In particular, embodiments of theinvention described in this specification are faster in overallperformance (i.e., categorizing several open-ended comments) and incategorizing each individual comment. Furthermore, the method of someembodiments involves lower costs of implementation and execution. Theimprovements over existing methods also include higher accuracy rates interms of correctly categorizing comments. In some cases, these benefitsare related to the manner in which the method allows a respondent whoprovides one or more comments to categorize the respondent's owncomments. This results in higher accuracy of the categorized comments.Other methods are too expensive and/or prone to categorization errors.By categorizing comments from a source (i.e., the respondent whooriginally made the comment(s)) with immediate and direct knowledge ofthe comment(s), the method captures the identified category immediatelyafter the comment is provided and with a high degree of certainty thatthe categorization is accurate.

The method of the present disclosure may be comprised of the followingsteps. This list of possible constituent steps is intended to beexemplary only and it is not intended that this list be used to limitthe method of the present application to just these steps. Personshaving ordinary skill in the art relevant to the present disclosure mayunderstand there to be equivalent steps that may be substituted withinthe present disclosure without changing the essential function oroperation of the method.

1. A comment in any form from any person is captured

2. A set of previous comments are selected

3. Based on categories entered by previous respondents, a list ofmost-likely categories is presented

4. The person who made the comment views the list and selects thecorrect category or categories or adds a new category

The method for categorizing open-ended comments of the presentdisclosure generally works according to the steps listed above so thatcorrect categories can be determined for future processing of comments.

By way of example, FIG. 1 conceptually illustrates a process in someembodiments for categorizing open-ended comments. The process willeither start with a pool of categories or not. In some embodiments, aresearcher or sponsor can enter a pool of categories before the processgets underway. But if the sponsor or researcher does not enter a pool ofcategories, the process ensues without any categories (at least until arespondent enters a category, as described below).

As shown in this figure, entering a pool of categories is adetermination made (at 105) before the categorization of open-endedcomments gets going. A research sponsor or researcher may specify a poolof categories because there may not be a list of categories from whichto select a category. Thus, the sponsor or researcher may enter (at 110)the pool of categories. The process then transitions to 115, which isdescribed below. On the other hand, the sponsor or researcher may notenter any categories. In this case, there may not be any categoriesentered before categorizing of open-ended comments gets started. Thenthe process simply moves on to receiving comments instead of enteringcategories at 110. A respondent would have to enter a category if nocategories have been entered by the sponsor or researcher at 110 or by aprevious respondent at 125.

The process 100 receives (at 115) a comment in response to an open-endedquestion. As noted above, in some embodiments a survey sponsor will posta question on an open forum for respondents to comment on. In order tocategorize the received comment, in some embodiments the process 100evaluates (at 120) the pool of categories in order to identify andpresent a category that is determined to be the most likely to match theopen-ended comment. In some embodiments, a set of likely categories areidentified and presented as possible matches to the received open-endedcomment.

Next, the process 100 receives a category pick (at 125) based on areview of the categories by respondent. In some embodiments, therespondent selects a set of categories for the comment. In someembodiments, if a matching category is not available for the respondentto select, a new category is entered. In some of these embodiments, therespondent inputs a name for the new category, which the process 100uses for the new category. In some embodiments, the process adds (at130) the new category to the pool of categories and assigns the commentto the new category.

After the received comment is categorized, the process 100 may repeat ifthere are more comments. Thus, the process determines (at 135) whetherthere is another comment to categorize. Specifically, for eachrespondent who makes a comment, the process is repeated starting from135 as described above. Otherwise, if there are no more respondents withcomments to categorize, the process 100 ends.

To make the method for categorizing open-ended comments of the presentdisclosure, a person would need to develop a software application toperform the above-listed steps. As the process 100 of some embodimentscan function in networked environments, any software applicationdeveloped for the process 100 should be included in a networked system.For example, a software application that acts as a client applicationmay be connected to a network, such as the Internet, and connect to aweb server in order to perform one or more of the steps of the process100.

By way of example, FIG. 2 conceptually illustrates a high levelschematic view of a system architecture that facilitates prioritizingcomments in some embodiments. As shown in this figure, the system 200includes a client computing device 210, a web server 220, and a cloudmanager application 230 operating on a back-end computing device. Insome embodiments, the client computing device 210 connects over anetwork to the web server 220. By connecting to the web server 220, theclient computing device can send and receive data. For instance, theclient computing device 210 may send requests to the web server 220 fordata that is provided by the cloud manager application 230. In someembodiments, the web server 220 processes requests for data that arereceived from client computing devices and communicates with the cloudmanager application 230 (over the network), to retrieve/receive therequested data. After receiving the requested data from the cloudmanager application 230, the web server 220 transmits the data to theappropriate requesting client computing device 210.

To use the method for categorizing open-ended comments of the presentdisclosure, a survey sponsor would need to determine when and where acomment is posted. Typically, a respondent in a research or othersetting will make an open-ended comment. Following the comment, therespondent will answer an additional question indicating the generalnature or category of their comment. The list of categories may bedifferent depending on the number of categories picked by others earlierin the research process. The best possible category is selected by theperson who made the original comment.

Additionally, in some embodiments the order could be changed so that theperson selects the category they want to comment on before the commentis made. In some circumstances, it is desirable to get the top-of-mindcomment before showing the person any categories which might affecttheir comment. Moreover, the method could be adapted for use in anyfield of technology that deals with open-ended comments in order togroup comments into categories.

While a system architecture is described by reference to FIG. 2, above,in some embodiments, the architecture of the system 200 includesmultiple client computing devices 210 and the cloud manager applicationrunning on the back-end computing device includes one or more databases,user and/or admin interfaces, and application programmer's interfaces(APIs).

FIG. 3 conceptually illustrates a detailed schematic view of a systemarchitecture that facilitates prioritizing comments in some embodiments.As shown in this figure, the system 300 includes a web server 310, acomputing device that provides an admin user interface 320, a computingdevice that provides one or more rest APIs 330, a computing device thatincludes application logic 340, a database that supports StructuredQuery Language (SQL) 350, a NoSQL database 355, and a plurality ofclient computing devices 360 a-360 c. Each of the client computingdevices 360 a, 360 b, and 360 c connects to the web server 310 over anetwork, such as the Internet, to send and receive data. In someembodiments, the data is retrieved from either or both databases 350 and355 via application logic 340 accessed through one or more of the restAPIs 330. In some embodiments, the rest API's are implemented on the webserver and data is retrieved via application logic 340 that isimplemented in a relational database management system (RDBMS). In thisway, the web server 310 is able to facilitate prioritizing comments fromany of several different client computing devices, including a browserapplication that operates on a computer 360 a (e.g., a desktop computer,a laptop computer, etc.), a client app operating on a mobile computingdevice 360 b, and a client app operating on a tablet computing device360 c.

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium or machine readable medium). When these instructions areexecuted by one or more processing unit(s) (e.g., one or moreprocessors, cores of processors, or other processing units), they causethe processing unit(s) to perform the actions indicated in theinstructions. Examples of computer readable media include, but are notlimited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc.The computer readable media does not include carrier waves andelectronic signals passing wirelessly or over wired connections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 4 conceptually illustrates an electronic system 400 with which someembodiments of the invention are implemented. The electronic system 400may be a computer, phone, PDA, or any other sort of electronic device.Such an electronic system includes various types of computer readablemedia and interfaces for various other types of computer readable media.Electronic system 400 includes a bus 405, processing unit(s) 410, asystem memory 415, a read-only 420, a permanent storage device 425,input devices 430, output devices 435, and a network 440.

The bus 405 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 400. For instance, the bus 405 communicativelyconnects the processing unit(s) 410 with the read-only 420, the systemmemory 415, and the permanent storage device 425.

From these various memory units, the processing unit(s) 410 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 420 stores static data and instructions thatare needed by the processing unit(s) 410 and other modules of theelectronic system. The permanent storage device 425, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system400 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 425.

Other embodiments use a removable storage device (such as a floppy diskor a flash drive) as the permanent storage device 425. Like thepermanent storage device 425, the system memory 415 is a read-and-writememory device. However, unlike storage device 425, the system memory 415is a volatile read-and-write memory, such as a random access memory. Thesystem memory 415 stores some of the instructions and data that theprocessor needs at runtime. In some embodiments, the invention'sprocesses are stored in the system memory 415, the permanent storagedevice 425, and/or the read-only 420. For example, the various memoryunits include instructions for processing appearance alterations ofdisplayable characters in accordance with some embodiments. From thesevarious memory units, the processing unit(s) 410 retrieves instructionsto execute and data to process in order to execute the processes of someembodiments.

The bus 405 also connects to the input and output devices 430 and 435.The input devices enable the user to communicate information and selectcommands to the electronic system. The input devices 430 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 435 display images generated by theelectronic system 400. The output devices 435 include printers anddisplay devices, such as cathode ray tubes (CRT) or liquid crystaldisplays (LCD). Some embodiments include devices such as a touchscreenthat functions as both input and output devices.

Finally, as shown in FIG. 4, bus 405 also couples electronic system 400to a network 440 through a network adapter (not shown). In this manner,the computer can be a part of a network of computers (such as a localarea network (“LAN”), a wide area network (“WAN”), or an intranet), or anetwork of networks (such as the Internet). Any or all components ofelectronic system 400 may be used in conjunction with the invention.

These functions described above can be implemented in digital electroniccircuitry, in computer software, firmware or hardware. The techniquescan be implemented using one or more computer program products.Programmable processors and computers can be packaged or included inmobile devices. The processes may be performed by one or moreprogrammable processors and by one or more set of programmable logiccircuitry. General and special purpose computing and storage devices canbe interconnected through communication networks.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For instance, FIG. 1 conceptuallyillustrates a process for categorizing open-ended comments. The specificoperations of the process 100 may not be performed in the exact ordershown and described. Specific operations may not be performed in onecontinuous series of operations, and different specific operations maybe performed in different embodiments. Furthermore, the process could beimplemented using several sub-processes, or as part of a larger macroprocess. Thus, one of ordinary skill in the art would understand thatthe invention is not to be limited by the foregoing illustrativedetails, but rather is to be defined by the appended claims.

I claim:
 1. A non-transitory computer readable medium storing a programwhich when executed by at least one processing unit of a computingdevice categorizes comments, said program comprising sets ofinstructions for: receiving a comment related to an open-ended question;evaluating a pool of categories to identify a set of categories to whichthe received comment is a likely match; receiving a selection of atleast one particular category from the identified set of categories; andcategorizing the received comment in the particular category.
 2. Thenon-transitory computer readable medium of claim 1, wherein the programfurther comprises a set of instructions for entering a set of categoriesto start the pool of categories.
 3. The non-transitory computer readablemedium of claim 1, wherein the received comment is associated with arespondent.
 4. The non-transitory computer readable medium of claim 3,wherein the set of instructions for receiving the selection of theparticular category comprises a set of instructions for receiving a userinput for the selection from the respondent.
 5. The non-transitorycomputer readable medium of claim 1, wherein the set of instructions forreceiving the selection of the particular category comprises a set ofinstructions for receiving a user input to enter a new category.
 6. Thenon-transitory computer readable medium of claim 5, wherein the selectedparticular category is the new category.
 7. The non-transitory computerreadable medium of claim 5, wherein the program further comprises a setof instructions for adding the new category to the pool of categories.8. The non-transitory computer readable medium of claim 1, wherein eachselected particular category is a category identified as a likely matchfor the received comment.
 9. The non-transitory computer readable mediumof claim 8, wherein each selection of a particular category is receivedas a selection from a respondent associated with the received comment.10. The non-transitory computer readable medium of claim 1, wherein theprogram further comprises a set of instructions for determining the nextrespondent to proceed with categorizing comments.